Live partition mobility using ordered memory migration

ABSTRACT

As disclosed herein a method, executed by a computer, for enabling live partition mobility using ordered memory migration includes receiving a request to initialize a migration of a logical partition (LPAR) to a destination system. The method further includes creating a list which includes memory page identifiers corresponding to memory pages of the LPAR. The memory page identifiers of the list are ordered according to a page transfer priority. The method further includes identifying memory pages of the LPAR that will be unmodified during an estimated duration of time of the migration. The method further includes updating the list, based on the identified memory pages of the LPAR that will be unmodified during the estimated duration of time of the migration. The method further includes migrating the LPAR based on the list. A computer system, and a computer program product corresponding to the method are also disclosed herein.

BACKGROUND

The present invention relates generally to system migration, and moreparticularly to system migration via live partition mobility usingordered memory migration.

A data center is a facility used to house computer systems andassociated components crucial to a company's vitality. Informationtechnology (IT) operations within a data center are a crucial aspect ofmany organizational operations within an industry. A company may operateits own data center, or optionally the company may receive its ITservices as a cloud computing service. Companies rely on their IToperations to run their day-to-day business activities, with a primaryconcern being maintaining business continuity. If a computer system orapplication becomes unavailable, company operations may be impaired orstopped completely. It is necessary to provide reliable infrastructurefor IT operations, in order to minimize any chance of servicedisruption. A data center must therefore keep high standards forassuring the integrity and functionality of its hosted computerenvironment.

The capacity of physical servers located in a data center is increasing,resulting in the number of virtual machines, also known as logicalpartitions or LPARs, defined on a physical server to also increase. AnLPAR is the division of a computer's processors, memory, and storageinto multiple sets of resources so that each set of resources can beoperated independently with its own operating system instance andapplications. Situations may arise (e.g., server maintenance) thatrequire a physical server to be shut down, meaning that all LPARs on thephysical server will have to be shut down as well. To avoid disruption,live partition mobility (LPM) may be used to migrate a running LPAR fromone managed system to another. LPM allows for the movement (i.e.,migration) of an active (i.e., running) LPAR from one system to anotherwith no application downtime.

SUMMARY

As disclosed herein a method, executed by a computer, for enabling livepartition mobility using ordered memory migration includes receiving arequest to initialize a migration of a logical partition (LPAR) to adestination system. The method further includes creating, by one or moreprocessors, a list, wherein the list includes one or more memory pageidentifiers corresponding to memory pages of the LPAR, wherein the oneor more memory page identifiers of the list are ordered according to apage transfer priority. The method further includes identifying, by oneor more processors, memory pages of the LPAR that will be unmodifiedduring an estimated duration of time of the migration. The methodfurther includes updating, by one or more processors, the list, basedon, at least, the identified memory pages of the LPAR that will beunmodified during the estimated duration of time of the migration. Themethod further includes migrating, by one or more processors, the LPARbased on, at least, the list. A computer system, and a computer programproduct corresponding to the method are also disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram depicting one embodiment of a virtualizedcomputing environment in which at least some of the embodimentsdisclosed herein may be deployed;

FIG. 2 is a flowchart depicting a method for memory page analysis, inaccordance with an embodiment of the present invention;

FIG. 3 is a flowchart depicting a method for detecting active processesthat have indicated they are about to exit (i.e., terminate execution),in accordance with an embodiment of the present invention;

FIG. 4 is a flowchart depicting a method for detecting inactiveprocesses that are waiting for the completion of some event beforeresuming activity, in accordance with an embodiment of the presentinvention;

FIG. 5 is a flowchart depicting a method for detecting memory pages thatmay be paged out of memory prior to migration completion, in accordancewith an embodiment of the present invention; and

FIG. 6 is a block diagram depicting various components of one embodimentof a computer suitable for executing the methods disclosed herein.

DETAILED DESCRIPTION

A data center may include many physical servers that are divided intoone or more logical partitions (LPARs) or virtual machines, with eachLPAR capable of providing a unique service. Services provided by an LPARmay include, but are not limited to, an AIX® server, a LINUX® server, ora virtual I/O server. It may be determined that an LPAR must be moved(i.e., migrated) to a different physical server without causing aninterruption of the service provided by the LPAR. The migration of theLPAR from a source server to an LPAR in a destination system may beperformed using live partition mobility (LPM) technology. LPM allows forthe migration of an active (e.g., running), suspended, or inactive(e.g., not activated) partition from one system to another with noapplication downtime.

LPM migration may comprise two phases, a pre-migration phase and anon-demand migration phase. In the pre-migration phase, memory pagescorresponding to an LPAR are copied asynchronously from a source systemto an LPAR on a destination, with dirty memory pages being re-copied asnecessary. Dirty memory pages are pages of memory whose contents on thesource system may have changed after the page was copied to thedestination system. The on-demand phase begins when a predefinedpercentage of the memory pages have been successfully copied to thedestination. Any remaining uncopied pages are copied to the destinationsystem synchronously (i.e., on-demand).

Embodiments of the present invention recognize that during the LPMpre-migration phase many pages are unnecessarily copied to thedestination system. For example, pages that will become dirty need notbe copied, pages that will be freed on the source system prior to theLPM migration completing need not be copied, pages that will be pagedout prior to the LPM migration completing need not be copied, and pagescorresponding to the LPM processes performing the migration need not becopied. However, migration processes unnecessarily transfer pagesmeeting the aforementioned scenarios. These issues, among others, causeadditional consumption of system resources which may delay thecompletion of the LPM migration operation and may cause the LPMmigration operation to time out. The embodiments disclosed hereingenerally address and solve the above-described problems.

FIG. 1 shows an example of virtualized computing environment 100applicable to various embodiments of the present invention. Virtualizedcomputing environment 100 includes a plurality of managed systems, suchas source system 110 and destination system 130. Also included invirtualized computing environment 100 is hardware management console(HMC) 160 that may be used to manage (e.g., configure and operate) oneor more LPARs (e.g., 112-1 to 112-n and 132-1 to 132-n) on managedsystems 110 and 130. Additionally, LPAR 112-2 includes migration process118. Migration process 118 may be a process within LPAR 112-2 that ismigrating LPAR 112-2 to another managed system (e.g., destination system130). In the depicted example, migration process 118 is coordinating themigration of LPAR 112-2 and corresponding memory area 122-2 to LPAR132-2 and memory area 142-2 on destination system 130.

Each of managed systems 110 and 130 and HMC 160 are connected to network170. Network 170 can be a local area network (LAN), a wide area network(WAN) such as the Internet, or a combination of the two, and can includewired, wireless, or fiber optic connections. In general, network 170 canbe any combination of connections and protocols that will supportcommunications between source system 110, destination system 130, andHMC 160, in accordance with embodiments of the invention. Virtualizedcomputing environment 100 may include additional computing devices orother devices not shown.

Source system 110 and destination system 130 each may be a laptopcomputer, tablet computer, netbook computer, personal computer (PC), adesktop computer, minicomputer mainframe computer or any programmableelectronic device capable of supporting virtualized computing. Sourcesystem 110 and destination system 130 can communicate with each other,HMC 160, and other computing devices (not shown) via network 170. Sourcesystem 110 and destination system 130 may also include internal andexternal hardware components, as depicted and described in furtherdetail with respect to FIG. 6.

LPARs (e.g., 112-1 to 112-n and 132-1 to 132-n) are logical partitionsof a computer (e.g. source system 110 and destination system 130) witheach partition appearing to have its own processors, memory, andstorage. Each LPAR can be operated independently running its ownoperating system and applications. In some embodiments, LPARs (e.g.,112-1 to 112-n and 132-1 to 132-n) may be configured as an AIX® server,a LINUX® server, a Virtual I/O server, or the like. In otherembodiments, LPARs (e.g., 112-1 to 112-n and 132-1 to 132-n) contain adatabase server, a file server, a mail server, a print server, a webserver, a gaming server, or an application server. In embodiments of theinvention, there may be any number (e.g., greater or fewer than depictedin FIG. 1) of LPARs 112-1 to 112-n and LPARs 132-1 to 132-n.

Memory areas (e.g., 122-1 to 122-n and 142-1 to 142-n) are logicalmemory mappings maintained by a hypervisor (e.g., hypervisors 120 and140). Source system 110 allows an LPAR (e.g., LPAR 112-1) use of aportion of physical memory corresponding to source system 110. LPAR112-1 will have a unique memory area (e.g., memory area 122-1) whichLPAR 112-1 references to access memory. In general, memory area 122-1 isa logical memory map, maintained by hypervisor 120, which identifies thephysical location of memory owned and used by LPAR 112-1, in accordancewith embodiments of the invention. In embodiments of the invention,there may be any number (e.g., greater or fewer than depicted in FIG. 1)of memory areas 122-1 to 122-n and memory areas 142-1 to 142-n.

HMC 160 may be a laptop computer, tablet computer, netbook computer,personal computer (PC), a desktop computer, or any programmableelectronic device capable of communicating with source system 110 anddestination system 130 via network 170. In general, HMC 160 representsany programmable electronic device or combination of programmableelectronic devices capable of executing machine readable programinstructions and communicating with other computing devices, such assource system 110, destination system 130, and other computing devices(not shown) via a network, such as network 170. HMC 160 may also includeinternal and external hardware components, as depicted and described infurther detail with respect to FIG. 6.

In one embodiment, one or more managed systems, such as source system110 and destination system 130, comprise LPARs (e.g. 112-1 to 112-n and132-1 to 132-n) and hypervisors 120 and 140. Hypervisors 120 and 140 maybe computer software, firmware, or hardware that creates and runsvirtual machines, such as LPARs 112-1 to 112-n and 132-1 to 132-n. Insome embodiment, hypervisor 120 allows multiple operating systems toshare a single hardware host (such as source system 110). In otherembodiments, hypervisor 120 controls the processor and resourcesallocated to each virtual system (e.g., 112-1 to 112-n) making sure thevirtual systems cannot disrupt each other.

Hypervisors 120 and 140 may also comprise mover services 124 and 144,and one or more memory areas (e.g., 122-1 to 122-n and 142-1 to 142-n).Mover services 124 and 144 may each be a program corresponding to an LPMoperation. In the depicted example, LPAR 112-2 and memory area 122-2 (ofsource system 110) are to be migrated to LPAR 132-2 and memory area142-2 (of destination system 130). Mover service 124 on source system110 sends memory pages corresponding to memory area 122-2 to moverservice 144 on destination system 130, and mover service 144 places thereceived memory pages in memory area 142-2. Mover service 124 transmitsthe memory pages to mover service 144 over network 170.

The embodiments disclosed herein may be leveraged by managed systems 110and 130 in order to facilitate improved performance during migration ofactive LPAR 112-2 between managed systems 110 and 130 using a generatedlist that identifies the order in which memory pages are to be moved.

FIG. 2 is a flowchart depicting a method for memory page analysis, inaccordance with an embodiment of the present invention. As depicted,memory page analysis method 200 includes receiving (210) a migrationinitialization request, creating (220) an in-core LPM list, identifying(230) writable memory pages, identifying (240) processes scheduled to bedispatched after LPM completion, identifying (250) system services thatare running, providing (260) the skipped-page list and priority-pagelist to a hypervisor, and spawning (270) processes to monitor processand memory activity. Memory page analysis method 200 enables systemmigration via LPM using ordered memory migration on source system 110.

Receiving (210) a migration initialization request may include sourcesystem 110 receiving, from HMC 160, a request to initiate an LPMmigration for LPAR 112-2 and corresponding memory area 122-2. Initiatinga migration may include starting an LPM migration process (e.g., adaemon) such as migration process 118 on LPAR 112-2. Initiating amigration may also include starting mover service 124 on hypervisor 120.Additionally, HMC 160 may supply, to both migration process 118 andmover service 124, the identity of a destination system (e.g.,destination system 130). Migration process 118 may identify LPAR 132-2and memory area 142-2 on destination system 130 as the target of the LPMmigration process. Mover service 124 may communicate with mover service144 on destination system 130, enabling LPAR 112-2 and memory area 122-2to be copied to LPAR 132-2 and memory area 142-2.

Creating (220) an in-core LPM list may include LPM migration process 118creating a list that contains page identifiers corresponding to memorypages currently present in a memory area 122-2 corresponding to LPAR112-2. In some embodiments, LPM migration process 118 sorts the memorypages identified in the in-core LPM according to a predeterminedtransfer order, producing an ordered in-core LPM list. Memory pagesdesignated as read-only pages cannot be updated, and therefore cannotbecome dirty if moved. In one embodiment, the transfer order of thememory pages (from highest to lowest priority) is identified asread-only pages, pinned pages, working storage pages, shared pages, filecache pages, and direct memory access (DMA) pages. Those of skill in theart will appreciate that there may be other logical orderingalternatives that may be used to produce an ordered in-core LPM list.LPM migration process 118 may provide the ordered LPM list to hypervisor120, enabling hypervisor 120 to begin transferring memory pages fromsource system 110 to destination system 130 according to the orderedin-core LPM list.

Additionally, LPM migration process 118 may also create a skipped-pagelist and a priority-page list. A skipped-page list may be used tocontain page identifiers of memory pages that should not be immediatelycopied to destination system 130. The priority-page list may be used tocontain page identifiers of memory pages that can be immediately copiedto destination system 130 with minimal probability of the pages becomingdirty during the migration operation. In some embodiments, LPM migrationprocess 118 uses the contents of the skipped-page list and priority-pagelist may to alter (e.g., reorder) the contents and order of in-core LPMlist.

Identifying (230) writable memory pages may include LPM migrationprocess 118 using system commands familiar to one of skill in the art toidentifying writable memory pages of currently running processescorresponding to LPAR 112-2. Active processes that own writeable memorypages continuously update the writable memory pages. If the writablememory pages are copied to destination system 130, the writeable memorypages may become dirty. To prevent copying dirty memory pages todestination system 130, LPM migration process 118 may add pageidentifiers corresponding to writable memory pages of currently runningprocesses to the skipped-page list.

Identifying (240) processes scheduled to be dispatched after LPMcompletion may include LPM migration process 118 using system commandsfamiliar to one of skill in the art to identify processes correspondingto LPAR 112-2 that are currently inactive, and scheduled to bedispatched (i.e., start running) after the estimated completion of theLPM migration. A process scheduled to be dispatched after the completionof the LPM migration will execute on destination system 130 only. Theidentified process will not execute source system 110, therefore, memorypages in memory area 122-2 corresponding to the identified process willnot be updated (i.e., will not become dirty) and can be migrated. Insome embodiments, LPM migration process 118 adds page identifiers, ofmemory pages corresponding to the identified process, to the prioritylist.

Identifying (250) system services that are running may include LPMmigration process 118 using system commands familiar to one of skill inthe art to identify system services and long running processes executingin an LPAR, for example, LPAR 112-2. System services may include, butare not limited to, system tuning processes, system tracing processes,system accounting processes, performance monitoring processes, and LPMmigration processes. In some embodiments, LPM migration process 118 addspage identifiers, of writable memory pages corresponding to the systemservices and long running processes, to the skipped-page list.

Providing (260) the skipped-page list and priority-page list to ahypervisor may include LPM migration process 118 updating (e.g.,reordering) the in-core LPM list with the contents of the skipped-pagelist and priority-page list. In some embodiments, LPM migration process118 produces an modified in-core LPM list by removing page identifiersof pages identified in the skipped-page list from the copy of thein-core LPM list and moving memory pages identified in the priority-pagelist to the top (e.g., highest priority) of the copy of the in-core LPMlist. LPM migration process 118 may provide the modified in-core LPMlist to hypervisor 120. In other embodiments, LPM migration process 118provides the skipped-page list and priority-page list to hypervisor 120,and hypervisor 120 modifies the current copy of the in-core LPM listusing the contents of the skipped-page list and priority-page lists,producing a modified in-core LPM list.

It should be noted that the in-core LPM list may be updated with thecontents of the skipped-page list and the priority-page list many timesduring a migration operation. In some embodiments, LPM migration process118 updates the in-core LPM list with the contents of the skipped-pagelist and the priority-page list immediately after the skipped-page listor the priority-page list is updated. In other embodiments, LPMmigration process 118 updates the in-core LPM list with the contents ofthe skipped-page list and the priority-page list at predeterminedintervals in the LPM migration process.

Spawning (270) monitoring processes to monitor LPAR process and memoryactivity may include LPM migration process 118 invoking one or moreprograms on source system 110. In some embodiments, the one or moreprograms execute as daemons running as background processes in an LPAR(e.g., LPAR 112-2). The monitoring processes may monitor process andmemory activity on an LPAR (e.g., LPAR 112-2) and memory area (e.g.,memory area 122-2). The monitoring processes may be used to monitorexiting (e.g., terminating) processes, monitor processes waiting for anevent to occur, monitor memory utilization, and the like. The monitoringprocesses may update the skipped-page list or the priority-page list,providing additional order and greater detail to the LPM migration. Themonitoring processes may continue to execute as a background process aslong as the pre-migration phase of an LPM migration is active. Once theLPM pre-migration process becomes inactive, all monitoring processescorresponding to the LPM migration may be terminated. A more detaileddescription of possible monitoring processes is provided in FIGS. 3-5.

FIG. 3 is a flowchart depicting an example of a possible program (ordaemon) that could be started as a result of the method depicted in FIG.2 (see step 270). The example program detects active processes that haveindicated they are about to exit (i.e., terminate execution), inaccordance with an embodiment of the present invention. As depicted,process exit tracker method 300 includes determining (310) whether aprocess is exiting, determining (320) whether a process is waiting on anevent, adding (330) pages to the skipped-page list, providing (340) theskipped-page list to a hypervisor, determining (350) whether LPMpre-migration phase is active, and performing (360) waiting processanalysis. Process exit tracker method 300 enables memory page analysismethod 200 to detect processes that may exit prior to the completion ofan LPM system migration. Exit tracker method 300 may be carried out byan exit detection operation.

Determining (310) whether a process is exiting may include the exitdetection operation monitoring a process exit flag, and detecting when aprocess is in an exit state (i.e., planning to exit). The exit flag maybe set in advance of the actual exit of the process. If the exitdetection operation detects that a process is in exit state (i.e., theprocess exit flag corresponding to the process is turned on), then theexit detection operation proceeds to determining (320) whether theprocess is waiting on an event. Otherwise, the exit detection operationproceeds to determining (350) whether the migration is complete.

Determining (320) whether a process is waiting on an event may includethe exit detection operation verifying if the process is currently in await state. A process in wait state may be waiting for an event to occurbefore the process can exit. If the exit detection operation detectsthat a process is waiting on an event, then the exit detection operationproceeds to performing (360) waiting process analysis. Otherwise, theexit detection operation proceeds to adding (330) pages to theskipped-page list.

Adding (330) pages to the skipped-page list may include the exitdetection operation retrieving page identifiers of pages correspondingto a process that may exit prior to completion of an LPM systemmigration. The exit detection operation may add the retrieved pageidentifiers to the skipped-page list. In some embodiments, a processexits (e.g., terminates) if the process runs to completion. In otherembodiments, a process terminates if the process encounters an errorsituation which results in an abnormal termination. The page identifiersof any memory pages corresponding to a process that may terminate priorto completion of an LPM system migration may be added to theskipped-page list.

Providing (340) the skipped-page list to a hypervisor may include theexit detection operation updating the in-core LPM list used by thehypervisor (e.g., hypervisor 120). Updating the in-core LPM list mayinclude removing from the in-core LPM list any pages identified in theskipped-page list. In some embodiments, the skipped-page list isprovided to hypervisor 120, and hypervisor 120 modifies the current copyof the in-core LPM list using the contents of the skipped-page list,producing a modified in-core LPM list. In other embodiments, the memorypages identified in the skipped-page list are removed from a copy of thein-core LPM list and the modified in-core LPM list is provided tohypervisor 120.

Determining (350) whether LPM pre-migration phase is active may includethe exit detection operation confirming that hypervisor 120 is activelycopying memory pages from a source system (e.g., source system 110) to adestination system (e.g., destination system 130). In some embodiments,the LPM pre-migration phase completes (i.e. becomes inactive) when apredefined percentage of the pages have been successfully copied todestination system 130. In other embodiments, the LPM pre-migrationphase becomes inactive when the LPM migration times out and terminates.If the exit detection operation detects that the LPM pre-migration isactive, then the exit detection operation continues monitoring forprocesses in an exit state and proceeds to determining (310) whether aprocess is exiting. Otherwise, the exit detection operation terminates.

Performing 360 waiting process analysis may include the exit detectionoperation determining what event must occur for the waiting process tocomplete the exit (e.g., termination) process. Waiting process analysismay be performed by waiting process tracker method 400. The waitingprocess analysis operation will be described in greater detail in thedescription of FIG. 4.

FIG. 4 is a flowchart depicting an example of a possible program (ordaemon) that could be started as a result of the method depicted in FIG.2 (see step 270). The example program detects inactive processes thatare waiting for the completion of some event before the waiting processcan resuming activity, in accordance with an embodiment of the presentinvention. As depicted, waiting process tracker method 400 includescalculating (410) a wake-up time, determining (420) whether a processwill run prior to LPM completion, adding (430) pages to the skipped-pagelist, providing (440) the skipped-page list to the hypervisor, adding(450) pages to the priority-page list, providing (460) priority-pagelist to the hypervisor, and determining (470) whether LPM pre-migrationphase is active. Waiting process tracker method 400 enables memory pageanalysis method 200 to detect inactive processes (e.g., processes in await state) that may not become active prior to the completion of theLPM system migration. Waiting process tracker method 400 may be carriedout by a wait state detection operation.

Calculating (410) a wake-up time may include the wait state detectionoperation determining a time when the event on which a process iswaiting is expected to complete, and the expected completion time may bedetermined to be the calculated wake-up time. In some embodiments, thewait state detection operation uses historical data corresponding to thecompletion time of the waited upon event to calculate a wake-up time. Inother embodiments the waiting process is waiting for a scheduled eventthat will not occur until after the LPM migration has completed, andtherefore the waiting process will not complete until after the LPMmigration has completed.

Determining (420) whether a process will run prior to LPM completion mayinclude the wait state detection operation comparing a calculatedwake-up time with the calculated LPM completion time (e.g., the LPMcompletion time determined in initializing (210) a migration operationof FIG. 2). If the wait state detection operation determines thecalculated wake-up time is prior to the calculated LPM completion time,then the wait state detection operation proceeds to adding (430) pagesto the skipped-page list. Otherwise the wait state detection operationproceeds to adding (450) pages to the priority-page list.

Adding (430) pages to the skipped-page list may include the wait statedetection operation continuing to process a waiting process (i.e.,processes in wait state) that may move out of wait state and executeprior to the completion of an LPM system migration. The page identifiersof any memory pages corresponding to a waiting process that may beginexecution prior to completion of a LPM system migration may be added tothe skipped-page list.

Providing (440) the skipped-page list to the hypervisor may include thewait state detection operation updating the in-core LPM list used by thehypervisor (e.g., hypervisor 120). Updating the in-core LPM list mayinclude removing from the in-core LPM list any pages identified in theskipped-page list. In some embodiments, the skipped-page list isprovided to hypervisor 120, and hypervisor 120 modifies the current copyof the in-core LPM list using the contents of the skipped-page list,producing a modified in-core LPM list. In other embodiments, the memorypages identified in the skipped-page list are removed from a copy of thein-core LPM list and the modified in-core LPM list is provided tohypervisor 120.

Adding (450) pages to the priority-page list may include the wait statedetection operation continuing to process a waiting processes (i.e.,processes in wait state) that may not move out of wait state prior tothe completion of a LPM system migration. The page identifiers of anymemory pages corresponding to a waiting process that may not executeprior to the completion of a LPM system migration may be added to thepriority-page list.

Providing (460) priority-page list to the hypervisor may include thewait state detection operation updating the in-core LPM list used by thehypervisor (e.g., hypervisor 120). In some embodiments, thepriority-page list is provided to hypervisor 120, and hypervisor 120modifies the current copy of the in-core LPM list using the contents ofthe priority-page list, producing a modified in-core LPM list. In otherembodiments, the memory pages identified in the priority-page list aremoved to the top (e.g., highest priority) of a copy of the in-core LPMlist, and the modified in-core LPM list is provided to hypervisor 120.

Determining (470) whether LPM pre-migration phase is active may includethe wait state detection operation confirming that hypervisor 120 isactively copying memory pages from a source system (e.g., source system110) to a destination system (e.g., destination system 130). In someembodiments, the LPM pre-migration phase completes (i.e. becomesinactive) when a predefined percentage of the pages have beensuccessfully copied to destination system 130. In other embodiments, theLPM pre-migration phase becomes inactive when the LPM migration timesout and terminates. If the wait state detection operation detects thatthe LPM pre-migration is active, then the wait state detection operationcontinues to monitor for processes in a wait state and proceeds tocalculating (410) a wake-up time. Otherwise, the wait state detectionoperation terminates.

FIG. 5 is a flowchart an example of a possible program (or daemon) thatcould be started as a result of the method depicted in FIG. 2 (see step270). The example program detects memory pages that may be paged out ofmemory prior to migration completion, in accordance with an embodimentof the present invention. As depicted, memory tracker method 500includes determining (510) whether a minimum free memory threshold hasbeen reached, adding (520) least recently used (LRU) pages to theskipped-page list, determining (530) whether page compression ordecompression is expected, adding (540) pages to the skipped-page list,providing (550) the skipped-page list to hypervisor 120, and determining(560) whether LPM pre-migration phase is active. Memory utilizationprocess tracker method 500 enables memory page analysis method 200 todetect memory pages that may page-out due to memory allocationrequirements prior to the completion of a LPM system migration. Memoryutilization process tracker method 500 may be carried out by a memorymonitoring operation.

Determining (510) whether a minimum free memory threshold has beenreached may include the memory monitoring operation using systemcommands to obtain current memory utilization statistics correspondingto the memory area that is being migrated (e.g., memory area 122-2 ofFIG. 1). The minimum free memory threshold may be a predetermined limitidentifying a minimum amount of free memory that is to be maintainedwithin a computing environment. Alternatively, the minimum free memorythreshold may be a system configurable limit identifying a minimumamount of free memory that is to be maintained within a computingenvironment. In some embodiments, the system commands corresponding tothe LPAR being migrated (i.e., LPAR 112-2) return statistics includingtotal memory, used memory, and free memory. In other embodiments, thesystem commands corresponding to the LPAR being migrated (i.e., LPAR112-2) return statistics including only total memory and used memory,allowing free memory to be calculated. If a minimum free memorythreshold has been reached, the memory monitoring operation proceeds toadding (520) least recently used (LRU) pages to the skipped-page list.Otherwise, the memory monitoring operation proceeds to determining (530)whether page promotion or demotion is expected.

Adding (520) least recently used (LRU) pages to the skipped-page listmay include the memory monitoring operation identifying which pages of amemory area (e.g., memory area 122-2 of FIG. 1) have been least recentlyused. LRU or cold pages may be pages in memory that are infrequentlyused. Those of skill in the art will appreciate that there are many waysto identify LRU or cold memory pages. The memory monitoring operationmay add page identifiers corresponding to LRU or cold pages to theskipped-page list. In some embodiments, the memory monitoring operationdetermines the number of pages to be added to the skipped-page list bycalculating the number of pages required to bring memory utilizationbelow the minimum free memory threshold. In other embodiments, apredetermined number of pages are repeatedly added to the skipped-pagelist until the minimum free memory threshold is no longer exceeded.

Determining (530) whether page compression or decompression is expectedmay include the memory monitoring operation determining if active memoryexpansion is enabled for the LPAR (i.e., LPAR 112-2) being migrated.Active memory expansion may enable pages that are infrequently used tobe compressed into a smaller space in memory, and when a compressed pageis referenced, it is decompressed (i.e., expanded) in memory. If thememory monitoring operation determines that page compression ordecompression is expected, the memory monitoring operation proceeds toadding (540) compressed or decompressed pages to the skipped-page list.Otherwise the memory monitoring operation continues monitoring memoryusage.

Adding (540) expected compressed or decompressed pages to theskipped-page list may include the memory monitoring operation placing,in the skipped-page list, page identifiers corresponding to memory pagesidentified as potential targets for page compression or decompression.Pages that are migrated to a destination system (e.g., destinationsystem 130) and then are compressed or decompressed on the source system(e.g., source system 110) can result in dirty memory pages that may haveto be copied to destination system 130 again.

Providing (550) the skipped-page list to hypervisor 120 may include thememory monitoring updating the in-core LPM list used by the hypervisor(e.g., hypervisor 120). Updating the in-core LPM list may includeremoving from the in-core LPM list any pages identified in theskipped-page list. In some embodiments, the skipped-page list isprovided to hypervisor 120, and hypervisor 120 modifies the current copyof the in-core LPM list using the contents of the skipped-page list,producing a modified in-core LPM list. In other embodiments, the memorypages identified in the skipped-page list are removed from a copy of thein-core LPM list and the modified in-core LPM list is provided tohypervisor 120.

Determining (560) whether LPM pre-migration phase is active may includethe memory monitoring operation confirming that hypervisor 120 isactively copying memory pages from a source system (e.g., source system110) to a destination system (e.g., destination system 130). In someembodiments, the LPM pre-migration phase completes (i.e. becomesinactive) when a predefined percentage of the pages have beensuccessfully copied to the destination system 130. In other embodiments,the LPM pre-migration phase becomes inactive when the LPM migrationtimes out and terminates. If the memory monitoring operation detectsthat the LPM pre-migration is active, then the memory monitoringoperation continues to monitor memory utilization and proceeds todetermining (510) whether a minimum free memory threshold has beenreached. Otherwise, the memory monitoring operation terminates.

FIG. 6 depicts a block diagram of components of a computer system 600,which is an example of a system such as source system 110, destinationsystem 130, and HMC 160 within virtualized computing environment 100 ofFIG. 1, in accordance with an embodiment of the present invention. Itshould be appreciated that FIG. 6 provides only an illustration of oneimplementation and does not imply any limitations with regard to theenvironments in which different embodiments can be implemented. Manymodifications to the depicted environment can be made.

Source system 110, destination system 130, and HMC 160 each includeprocessor(s) 604, cache 614, memory 606, persistent storage 608,communications unit 610, input/output (I/O) interface(s) 612 andcommunications fabric 602. Communications fabric 602 providescommunications between cache 614, memory 606, persistent storage 608,communications unit 610, and input/output (I/O) interface(s) 612.Communications fabric 602 can be implemented with any architecturedesigned for passing data and/or control information between processors(such as microprocessors, communications and network processors, etc.),system memory, peripheral devices, and any other hardware componentswithin a system. For example, communications fabric 602 can beimplemented with one or more buses.

Memory 606 and persistent storage 608 are computer readable storagemedia. In this embodiment, memory 606 includes random access memory(RAM). In general, memory 606 can include any suitable volatile ornon-volatile computer readable storage media. Cache 614 is a fast memorythat enhances the performance of processor(s) 604 by holding recentlyaccessed data, and data near recently accessed data, from memory 606.

Program instructions and data used to practice embodiments of thepresent invention, e.g., memory page analysis method 200, are stored inpersistent storage 608 for execution and/or access by one or more of therespective processor(s) 604 via cache 614. In this embodiment,persistent storage 608 includes a magnetic hard disk drive.Alternatively, or in addition to a magnetic hard disk drive, persistentstorage 608 can include a solid-state hard drive, a semiconductorstorage device, a read-only memory (ROM), an erasable programmableread-only memory (EPROM), a flash memory, or any other computer readablestorage media that is capable of storing program instructions or digitalinformation.

The media used by persistent storage 608 may also be removable. Forexample, a removable hard drive may be used for persistent storage 608.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer readable storage medium that is also part of persistent storage608.

Communications unit 610, in these examples, provides for communicationswith other data processing systems or devices, including resources ofsource system 110, destination system 130, and HMC 160. In theseexamples, communications unit 610 includes one or more network interfacecards. Communications unit 610 may provide communications through theuse of either or both physical and wireless communications links.Program instructions and data used to practice embodiments of memorypage analysis method 200 may be downloaded to persistent storage 608through communications unit 610.

I/O interface(s) 612 allows for input and output of data with otherdevices that may be connected to each computer system. For example, I/Ointerface(s) 612 may provide a connection to external device(s) 616 suchas a keyboard, a keypad, a touch screen, a microphone, a digital camera,and/or some other suitable input device. External device(s) 616 can alsoinclude portable computer readable storage media such as, for example,thumb drives, portable optical or magnetic disks, and memory cards.Software and data used to practice embodiments of the present inventioncan be stored on such portable computer readable storage media and canbe loaded onto persistent storage 608 via I/O interface(s) 612. I/Ointerface(s) 612 also connect to a display 618.

Display 618 provides a mechanism to display data to a user and may be,for example, a computer monitor.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

1-7. (canceled)
 8. A computer program product for enabling livepartition mobility using ordered memory migration, the computer programproduct comprising: one or more computer readable storage media andprogram instructions stored on the one or more computer readable storagemedia, the program instructions comprising instructions to: receive arequest to initialize a migration of a logical partition (LPAR) to adestination system; create a list, wherein the list comprises one ormore memory page identifiers corresponding to memory pages of the LPAR,wherein the one or more memory page identifiers of the list are orderedaccording to a page transfer priority; identify memory pages of the LPARthat will be unmodified during an estimated duration of time of themigration; update the list, based on, at least, the identified memorypages of the LPAR that will be unmodified during the estimated durationof time of the migration; and migrate the LPAR based on, at least, thelist.
 9. The computer program product of claim 8, wherein the programinstructions to identify the memory pages of the LPAR that will beunmodified during the estimated duration of time of the migrationcomprise: instructions to identify memory pages corresponding to aprocess scheduled to dispatch after an estimated completion time of themigration; and instructions to identify memory pages designated asread-only.
 10. The computer program product of claim 8, the programinstructions further comprising: instructions to identify writeablememory pages corresponding to a currently running process of the LPAR;and instructions to update the list further based on the identifiedwriteable memory pages corresponding to the currently running process ofthe LPAR.
 11. The computer program product of claim 10, wherein theprogram instructions to update the list comprise: instructions toreorder the list such that memory page identifiers corresponding to theidentified memory pages corresponding to the currently running processof the LPAR follow other memory page identifiers of the list.
 12. Thecomputer program product of claim 8, wherein the program instructions toupdate the list comprise: instructions to reorder the list such thatmemory page identifiers corresponding to the identified memory pages ofthe LPAR that will be unmodified during the estimated duration of timeof the migration precede other memory page identifiers of the list. 13.The computer program product of claim 8, the program instructionsfurther comprising: instructions to identify memory pages expected to becompressed during the estimated duration of time of the migration, basedon, at least, infrequency of use of the memory pages; and instructionsto update the list further based on the identified memory pages expectedto be compressed during the estimated duration of time of the migration.14. The computer program product of claim 8, wherein the page transferpriority includes, at least, an ordered identification of read-onlypages, pinned pages, working storage pages, shared pages, file cachepages, and direct memory access pages.
 15. A computer system forenabling live partition mobility using ordered memory migration, thecomputer system comprising: one or more computer processors, one or morecomputer readable storage media, and program instructions stored on theone or more computer readable storage media for execution by at leastone of the one or more processors, the program instructions comprisinginstructions to: receive a request to initialize a migration of alogical partition (LPAR) to a destination system; create a list, whereinthe list comprises one or more memory page identifiers corresponding tomemory pages of the LPAR, wherein the one or more memory pageidentifiers of the list are ordered according to a page transferpriority; identify memory pages of the LPAR that will be unmodifiedduring an estimated duration of time of the migration; update the list,based on, at least, the identified memory pages of the LPAR that will beunmodified during the estimated duration of time of the migration; andmigrate the LPAR based on, at least, the list.
 16. The computer systemof claim 15, wherein the program instructions to identify the memorypages of the LPAR that will be unmodified during the estimated durationof time of the migration comprise: instructions to identify memory pagescorresponding to a process scheduled to dispatch after an estimatedcompletion time of the migration; and instructions to identify memorypages designated as read-only.
 17. The computer system of claim 15, theprogram instructions further comprising: instructions to identifywriteable memory pages corresponding to a currently running process ofthe LPAR; and instructions to update the list further based on theidentified writeable memory pages corresponding to the currently runningprocess of the LPAR.
 18. The computer system of claim 17, wherein theprogram instructions to update the list comprise: instructions toreorder the list such that memory page identifiers corresponding to theidentified memory pages corresponding to the currently running processof the LPAR follow other memory page identifiers of the list.
 19. Thecomputer system of claim 15, wherein the program instructions to updatethe list comprise: instructions to reorder the list such that memorypage identifiers corresponding to the identified memory pages of theLPAR that will be unmodified during the estimated duration of time ofthe migration precede other memory page identifiers of the list.
 20. Thecomputer system of claim 15, the program instructions furthercomprising: instructions to identify memory pages expected to becompressed during the estimated duration of time of the migration, basedon, at least, infrequency of use of the memory pages; and instructionsto update the list further based on the identified memory pages expectedto be compressed during the estimated duration of time of the migration.